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ABSTRACT 

In  recent  years,  the  U.S.  Marine  Corps  has  begun  developing 
an  infrastructure  for  applying  agent-based  models  and  simu¬ 
lation,  computing  power,  and  data  analysis  and  visualization 
technologies  to  help  answer  complex  questions  in  military 
operations.  Factor  screening  approaches  are  of  particular 
interest,  since  even  relatively  simple  agent-based  models 
may  have  hundreds  (or  even  thousands)  of  inputs  that  can 
be  varied.  We  describe  a  new  experimental  design,  called 
a  frequency-based  design,  that  can  be  used  for  exploring 
the  behavior  of  terminating  simulations.  We  apply  this  to 
a  model  of  a  peace-enforcement  operation.  We  examine 
the  behavior  of  four  performance  measures  (including  two 
attrition  ratios)  and  discuss  how  the  results  confirm  and 
complement  earlier  findings.  We  conclude  with  a  brief 
discussion  of  issues  that  merit  further  investigation. 

1  INTRODUCTION 

Our  motivation  for  exploring  this  approach  arose  out  of 
work  we  have  been  doing  for  the  U.S.  Marine  Corps  under 
their  “Project  Albert”  umbrella.  This  multi-year,  multi¬ 
national  effort  attempts  to  exploit  advances  in  three  core 
disciplines:  1)  Agent-based  models  and  simulations;  2) 
Computing  power;  and  3)  Data  visualization.  Data  farming 
(Brandstein  and  Horne,  1998)  is  the  application  of  these 
disciplines  to  help  answer  complex  questions  in  military 
operations.  The  four  principal  processes  of  data  farming 
are  fertilization,  cultivation,  planting,  and  harvesting.  Fer¬ 
tilization  means  providing  military  professionals  and  other 
experts  with  ideas  on  how  to  capture  important  aspects  of 
conflict  that  have  not  been  taken  into  account  in  the  past, 
such  as  morale,  leadership,  timing,  intuition,  adaptability, 
etc.  Cultivation  means  receiving  ideas  from  these  profes¬ 
sionals  about  what  might  be  important  in  a  given  situation. 
Planting  means  incorporating  these  ideas  into  models,  to  the 
extent  possible,  and  running  the  models  over  a  wide  variety 


of  possibilities  by  varying  simulation  parameters  (also  called 
factors).  Finally,  innovative  techniques  for  understanding 
scientific  data  are  used  when  harvesting  the  model  output. 
Just  as  the  farmer  grows  crops  to  meet  the  needs  of  the 
consumers,  who  are  hungry  for  food,  the  data  farmer  grows 
data  to  meet  the  needs  of  the  ultimate  decision-makers,  who 
are  hungry  for  answers. 

The  propensity  to  produce  large  multi-dimensional  data 
sets  with  several  measures  of  performance  (MOPs)  is  in¬ 
herent  in  data  farming.  However,  care  must  be  taken  when 
generating  data,  because  the  time  required  to  examine  all 
potential  factor  level  combinations  grows  exponentially  with 
the  number  of  factors  investigated.  Furthermore,  because 
Project  Albert  uses  agent-based  simulations  to  model  mili¬ 
tary  operations,  some  of  the  factors  are  only  notional  repre¬ 
sentations  of  human  thought  or  behavior — such  as  morale, 
unit  discipline,  leadership,  or  aggressiveness.  The  data 
farming  environment  is  thus  best  viewed  as  one  that  may 
provide  the  decision-makers  with  qualitative  insights  rather 
than  numerical  predictions  (Lucas  et  ah,  2002,  2003). 

The  vagueness  associated  with  interpreting  the  factors 
can  complicate  the  task  of  capturing  and  communicating 
the  essence  of  the  data  set.  However,  gaining  insight  into 
the  model’s  behavior  is  challenging  for  any  simulation  with 
a  large  number  of  factors,  particularly  if  interaction  effects 
among  two  or  more  of  these  factors  are  possible.  Brute 
force  methods  are  not  practical  unless  the  number  of  factors 
is  small.  Trial-and-error  methods  are  unlikely  to  provide  the 
analyst  with  an  understanding  of  how  the  MOPs  are  affected 
by  the  input  factor  settings.  Instead,  systematic  experimental 
designs  are  needed  in  order  to  efficiently  generate  data  that 
can  be  used  to  provide  insights  into  the  model’s  behavior 
and  guide  further  investigations. 

The  type  of  design  that  is  most  appropriate  depends 
on  both  the  number  of  factors  and  the  nature  of  the  re¬ 
sponse  surface.  For  an  overview  of  the  possibilities,  we 
refer  the  reader  to  Sanchez  and  Lucas  (2002)  or  Kleijnen 
et  al.  (2003).  Clearly,  no  single  design  is  best  for  all 
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situations.  However,  the  ability  to  use  prior  information 
can  dramatically  reduce  the  need  for  additional  runs.  This 
means  that  sequential  designs  have  distinct  advantages  over 
non-sequential  designs. 

In  this  paper,  we  propose  a  frequency-based  design 
(FBD)  that  is  appropriate  for  analyzing  terminating  simu¬ 
lations.  We  present  some  background  material  on  spectral 
analysis  in  Section  2,  and  describe  FDB  in  Section  3.  An 
agent-based  implementation  of  a  peace-enforcement  sce¬ 
nario  is  explored  using  FBD  in  Section  4.  We  conclude 
with  a  brief  discussion  of  some  issues  that  merit  further 
investigation. 

2  BACKGROUND 


In  this  section  we  present  a  very  brief  overview  of  some 
important  concepts  in  spectral  analysis.  We  also  summarize 
a  technique  called  frequency  domain  experimentation,  which 
was  developed  by  Schruben  and  Cogliano  (1987)  as  a  factor¬ 
screening  approach  for  non-terminating  simulations. 

2.1  Spectral  Analysis 


Spectral  analysis  has  long  been  used  for  identifying  the 
cyclic  components  of  time-series  data  (Chatfield,  1996). 

Let  Y  =  [yi  y2  ...  y,v]T  be  an  indexed  set  of  ob¬ 
servations.  For  notational  convenience,  assume  N  is 
even  and  let  H  =  N/ 2.  We  can  decompose  Y  into 
its  cyclic  components  by  frequency  as  Y  =  A 9,  where 
9  =  [fi  a i  ySi  ...  an  fin  <*N/ 2]  and  A  is  the  matrix 


( 1  ...  cos  («,-)  sin(o),) 
1  ...  cos (2 a>i)  sin (2a),) 


cos(7r)  \ 
cos(27r) 


■  (1) 


\1  ...  cos(Aa)/)  sin(Aa>;)  ...  cos(ALr) / 


The  pairs  of  cosine  and  sine  terms  correspond  to  a>/  = 
2ni /N  for  i  =  1,  . . . ,  H .  These  frequencies  are  expressed 
in  radians  per  observation,  e.g.,  an  oscillation  of  one  cycle  per 
observation  equals  2n  radians  per  observation.  Furthermore, 
for  any  discretely-sampled  signal  it  is  sufficient  to  display 
only  frequencies  ranging  from  [0, n]  in  the  spectrum  because 
the  highest  observable  frequency  is  n  radians  per  cycle 
(equivalently,  one-half  cycle  per  observation). 

The  Fourier  transform  of  Y  is  9  =  (AtA)-1AtT. 
Moving  from  the  Fourier  transform  to  the  (Fourier)  spectrum 
involves  squaring  the  estimated  coefficients  for  the  sine 
and  cosine  terms,  and  summing  them  by  frequency.  The 
spectrum  components  are  thus  (af  +  Br)  for  i  =  1 ,...,// 
and  aj  for  i  —  N / 2.  It  can  be  demonstrated  that 

1  N  N/2 

„  !](>')  -  yf  =  (ai  +  A2)  +  4/2-  (2) 

V  !=1  !=1 


i.e.,  the  Fourier  spectrum  partitions  the  variance.  Under 
mild  assumptions  (Chatfield,  1996),  the  estimated  spectral 
coefficients  have  a  chi-squared  distribution. 

While  the  regression  representation  makes  it  clear  how 
the  partitioning  works,  other  computationally  efficient  meth¬ 
ods  can  be  used  to  estimate  the  spectrum.  The  Wiener- 
Khintchine  theorem  relates  the  Fourier  spectrum  of  the 
model  to  the  Fourier  transform  of  the  autocovariance  func¬ 
tion  of  the  observations  in  the  data  set.  Autocovariance  is 
a  measure  of  the  covariance  of  a  sequence  of  observations 
with  each  other.  For  stationary  processes,  autocovariance 
generally  diminishes  as  the  observations  become  sufficiently 
far  apart.  A  technique  called  windowing  uses  only  a  speci¬ 
fied  number  of  observations  (M,  called  the  window  size)  to 
estimate  the  spectrum  efficiently.  A  common  choice  is  to 
select  M  to  be  proportional  to  \f~N ,  although  other  values 
of  M  are  possible  so  long  as  that  the  ratio  M/N  — >  0  as 
M,  N  —>  oc.  A  more  thorough  explanation  of  Fourier 
analysis  can  be  found  in  Chapter  7  of  Chatfield  (1996). 

One  other  important  issue  is  frequency  aliasing.  If 
a  frequency  oscillates  at  a  rate  higher  than  one-half  cycle 
per  observation,  it  appears  to  be  “folded"  back  into  the 
range  [0,  n]  when  sampled  discretely.  Figure  1  illustrates 
this  behavior.  Both  signals  completes  10  full  cycles  in  the 
graph.  The  sampling  rate  in  the  upper  figure  is  high  enough 
(1/12  cycle  per  observation)  that  an  observer  would  view 
the  true  frequency.  However,  aliasing  occurs  in  the  lower 
graph  where  the  sampling  rate  is  5/3  cycles  per  observation. 
The  sampled  data  appear  to  complete  only  two  cycles,  for 
an  apparent  frequency  of  1/3  cycle  per  observation. 

2.2  Frequency  Domain  Experiments  for 

Non-Terminating  Simulations 

The  idea  of  oscillating  input  factor  levels  was  first  explored 
by  Schruben  and  Cogliano  (1987).  Their  approach,  called 
frequency  domain  experimentation  (FDE),  investigates  the 
impact  of  several  input  factors  on  system  performance  by 
varying  input  factor  levels  within  the  course  of  a  single,  very 
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long  run  called  the  signal  run.  All  input  factors  are  held 
constant  at  nominal  levels  during  another  run  of  the  same 
length,  called  the  noise  run.  After  truncating  both  the  signal 
and  the  noise  runs  to  remove  the  initial  transients,  the  Fourier 
spectra  of  the  two  output  streams  are  obtained.  The  ratios  of 
the  signal  spectrum  to  the  noise  spectrum  are  computed  by 
frequency.  High  values  or  “spikes”  in  these  signal-to-noise 
ratios  (SNRs)  indicate  that  the  associated  input  term  is  an 
important  contributor  to  the  output  behavior. 

The  frequency  at  which  a  factor  is  varied  is  called  its 
driving  frequency.  In  the  frequency  domain,  the  spectrum 
displays  all  frequencies  contributing  to  the  variations  in  the 
response.  Furthermore,  the  indicator  frequencies  for  higher- 
order  effects  of  the  oscillated  factors  on  response  show  up 
at  well-defined  locations.  For  example,  suppose  factor  X\ 
is  assigned  a  driving  frequency  co\ .  The  main  effect  of  X  \ 
on  the  response  spectrum  has  an  indicator  frequency  of  a>\. 
If  factor  X\  has  a  quadratic  effect  on  the  response,  this  is 
associated  with  an  indicator  frequency  of  2a>\.  Similarly, 
the  nth-order  effect  of  X\  on  the  response  has  an  indicator 
frequency  of  na>\  (or  its  alias)  in  the  response  spectrum. 
For  interaction  terms,  the  indicator  frequencies  are  the  sums 
and  differences  of  the  driving  frequencies.  For  example, 
suppose  there  is  a  second-order  interaction  effect  on  the 
response  from  two  factors,  X\  and  Xi.  where  X \  and  Xi 
are  assigned  driving  frequencies  of  an  and  an ,  respectively. 
The  second-order  interaction  term  in  the  response  has  two 
indicator  frequencies:  o>\  +  on  and  o>\  —  a> 2,  respectively.  If 
an  +a>2  falls  outside  the  interval  [0,  2: r],  then  the  associated 
indicator  frequency  is  the  alias  (and  similarly  for  an  —  anf. 

Sanchez  and  Buss  (1987)  provide  a  mathematical  model 
formulation  which  explains  how  and  why  FDEs  work.  Ja¬ 
cobson,  Buss  and  Schruben  (1991)  provide  an  algorithm 
which  assigns  driving  frequencies  to  the  factors  to  allow 
investigation  of  full  second-order  or  third-order  models. 
Sanchez  and  Konana  (2000)  investigate  how  the  total  data 
collection  effort  should  be  allocated  between  the  signal  and 
noise  runs,  with  the  goal  of  improving  the  efficiency  of 
FDEs. 

3  FREQUENCY-BASED  DESIGNS  FOR 
TERMINATING  SIMULATIONS 

We  now  describe  a  new  data-farming  approach:  the  use 
of  frequency-based  designs  (FBDs)  for  terminating  simula¬ 
tions.  While  we  make  use  of  many  of  the  building  blocks 
of  FDE,  the  FDE  approach  cannot  be  directly  applied  to 
terminating  simulations.  For  some  simulation  models,  such 
as  the  peace-enforcement  application  of  Section  4,  long  runs 
are  not  possible.  For  others,  such  as  rare-event  simulation, 
the  time  until  termination  may  itself  be  the  performance 
measure. 

Our  frequency-based  designs  are  similar  to  the  signal 
runs  for  FDEs  in  terms  of  factor  level  selection,  etc.  They 


differ  in  several  notable  ways.  First,  the  input  factor  levels 
are  held  constant  for  the  duration  of  each  run,  and  we  obtain 
only  one  data  summary  from  each  run  that  will  be  used 
for  analysis  purposes.  Second,  all  runs  are  independently 
seeded — which  means  there  are  no  serially  correlated  errors 
in  the  run-to-run  results.  Third,  since  there  are  no  concerns 
about  initial  bias,  we  can  use  all  of  the  data  generated  by 
the  simulation.  Fourth,  our  implementation  exploits  the 
power  of  supercomputers  by  parallelizing  the  data  farming 
process.  While  this  last  point  is  not  a  requirement  for  using 
FBDs,  it  does  have  practical  benefits  by  reducing  the  total 
length  of  time  necessary  to  acquire  the  data. 

We  now  provide  enough  detail  for  the  interested  reader 
to  generate  and  apply  FBDs  to  terminating  simulations. 
Alternatively,  software  programs  to  perform  the  design  and 
analysis  are  available  (Sanchez,  2002).  Our  example  as¬ 
sumes  we  are  interested  in  fitting  a  second-order  model, 

i.e.,  one  where  quadratic  and/or  two-way  interactions  may 
be  present.  A  similar  approach  can  be  used  for  higher-order 
models,  although  the  required  number  of  runs  increases. 

Suppose  there  are  k  input  factors  X  \ , ...  ,X^  we  wish 
to  investigate.  To  specify  the  design  we  must  determine 
the  number  of  runs  ( N )  and  the  factor  levels  at  each  run 
(Xjj,  i  =  \, ...  ,k\  t  =  ,  N).  Our  procedure  follows. 

1.  Specify  the  input  factors  X[ , . . . ,  X%,  along  with 
middle,  minimum  and  maximum  values  of  interest 
for  the  experiment  (m,,  m,  —  a,-,  m +  a/,  respec¬ 
tively). 

2.  Determine  a  set  of  driving  frequencies  a>i, ...  ,a>k 
that  will  allow  identification  of  any  important  terms 
in  a  2nd  order  metamodel: 

k  k  k 

Yt  =  A) + y '  ^  Xi.t  +  y '  y '  Pij.tXi^xjt. 
i= 1  i=l  j=i 

Let  /  denote  the  lowest  common  denominator 
of  the  frequencies  when  expressed  in  cycles  per 
observation,  and  let  p  =  2k+[k(k— 1)/2]  represent 
the  total  number  of  terms  in  the  full  model. 

3.  Set  N  =  cf  for  some  number  of  replications  c. 
Then  set 

Xj  t  =  nij  +  at  cos  (a>i  t ) ,  t  =  1, . . . ,  N . 

for  i  =  1 , ...  ,k  and  t  =  1 , . . . ,  N.  Y,  is  the  output 
of  run  t. 

4.  Take  the  Fourier  transform  of  the  output  Y  to  obtain 
0.  Under  mild  assumptions,  the  components  of  the 
spectrum  will  have  chi-squared  distributions  (let  v 
denote  the  degrees  of  freedom).  The  variance 
in  Y  is  now  partitioned  into  components  for  the 
frequencies.  Let  So  denote  the  spectrum  evaluated 
at  frequency  a>  (or  its  alias  in  the  range  [0,  zr  ] . 
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5.  Sum  the  spectral  values  at  all  non-indicator  terms 
to  obtain  an  estimate  of  the  noise  component.  Call 
this  value  V.  The  number  of  df  associated  with  V 
is  r  =  v  [0.5(/  -  1)  -  p], 

6.  Compute  signal-to-noise  ratios  (SNRs)  by  mapping 
the  indicator  frequencies  to  the  metamodel  terms 
and  dividing  by  the  noise  component. 

•  For  the  main  effect  of  X;, 


for  each  experiment,  It  follows  from  the  Wiener-Khintchine 
theorem  (Chatfield,  1996)  that  the  spectrum  of  independent 
observations  is  flat.  Thus,  under  the  null  hypothesis  that 
there  are  no  factor  effects,  the  heights  of  the  indicator  and 
non-indicator  frequencies  have  the  same  expected  value  for 
all  co,  and  so  the  expected  value  of  each  SNR  is  equal  to  1. 

4  A  PEACE-ENFORCEMENT  EXAMPLE 


SNR  = 

V/r 

For  the  quadratic  effect  Xj, 

S2u>i  /  V 


SNR  = 


V/r 


For  the  interaction  effect  of  X,  and  X j , 

(SfOj+Qjj  -f  S(0[  —o)j ) / 2 u 


SNR  = 


V/r 


7.  Graph  or  tabulate  the  SNRs. 

8.  For  an  overall  test  at  level  a,  compare  the  SNRs 
for  the  main  and  quadratic  effects  to  the  critical 
value  r^p) .  and  compare  the  SNRs  for  the 

interaction  effects  to  the  critical  value  F^v^p\ 
Alternatively,  particularly  if  many  SNRs  are  statis¬ 
tically  significant,  a  qualitative  assessment  identi¬ 
fies  those  terms  with  the  largest  SNRs  as  the  most 
important. 

9.  If  no  terms  are  statistically  significant,  either  stop 
(and  conclude  that  the  selected  input  factors  do  not 
affect  the  response  over  the  ranges  examined)  or 
increase  c  and  go  back  to  Step  3  to  collect  more 
data. 

Software  useful  for  designing  and  analyzing  FBDs  (or 
FDEs)  is  available  in  the  Java©  programming  language 
(Sanchez,  2002).  The  Design  program  requires  the  number 
of  factors  as  the  input,  and  returns  a  set  of  frequency 
assignments  as  in  Table  1.  The  Fourier  program  requires 
the  following  inputs:  the  number  of  frequencies  into  which 
the  response  is  to  be  partitioned,  the  window  size,  the 
type  of  windowing,  and  the  number  of  observations  in  the 
input  data  set.  The  program  then  estimates  the  spectrum  of 
the  observations  and  produces  a  response  spectrum.  The 
program  automatically  adds  one  more  partition  for  the  zero 
frequency,  that  corresponds  to  the  constant  term  in  the 
regression  model.  Thus,  the  spectral  power  at  the  zero 
frequency  signifies  the  contribution  of  the  constant  term  in 
the  regression  model  to  the  response. 

Note  that  the  experimental  units  are  independent,  by 
construction,  since  we  use  different  random  number  seeds 


Peace  enforcement  is  a  critical  component  of  current  and 
future  military  operations.  According  to  the  U.S.  Army 
Field  Manual  100-23  (Department  of  the  Army,  1994),  peace 
enforcement  is  “the  application  of  military  force  or  the  threat 
of  its  use,  normally  pursuant  to  international  authorization, 
to  compel  compliance  with  generally  accepted  resolutions  or 
sanctions.  The  purpose  of  peace  enforcement  is  to  maintain 
or  restore  peace  and  support  diplomatic  efforts  to  reach  a 
long-term  political  settlement." 

There  are  many  ways  in  which  peace-enforcement  op¬ 
erations  can  be  investigated.  We  consider  a  scenario  devel¬ 
oped  by  Cioppa  (2002),  who  developed  and  used  nearly- 
orthogonal  Latin  hypercubes  to  explore  the  model’s  perfor¬ 
mance.  The  scenario  was  deemed  doctrinally  correct  and 
plausible  by  the  U.S.  Army  Infantry  Simulation  Center  at 
Fort  Benning,  Georgia.  The  scenario  was  implemented  in 
MANA  (Map  Aware  Non-uniform  Automata) — an  agent- 
based  modeling  platform  that  was  developed  for  the  New 
Zealand  Army  and  Defence  Force  (Stephen  and  Lauren, 
2001).  It  has  a  graphical  user-interface  for  specifying  ini¬ 
tial  conditions  and  trigger  states  of  the  agents,  as  well  as  for 
animating  the  simulation  run.  MANA  also  offers  the  user 
the  ability  to  specify  levels  of  input  parameters  easily  from 
a  formatted  input  file.  MANA  is  one  of  Project  Albert’s 
data-farmable  suite  of  modeling  platforms. 

4.1  Scenario 

Figure  2  illustrates  the  initial  positions  of  the  agents  for  a 
single  run  of  the  software.  The  ellipses  indicate  four  areas 
of  operations  (AOs).  Clockwise  from  the  center  left  are 
AO  Rattler,  AO  Python,  AO  Cobra  and  AO  Boa.  Blue’s 
mission  is  to  clear  AO  Cobra  within  the  next  two  hours 
in  order  to  facilitate  United  Nations  (UN)  food  distribution 
and  military  convoy  operations.  Blue  uses  a  light  infantry 
platoon  composed  of  three  nine-man  rifle  squads  and  a 
platoon  headquarters  (HQ)  of  seven  soldiers  containing  two 
machine  guns.  Their  movement  scheme  is  one  squad  up 
and  two  squads  back,  with  the  platoon  HQ  following  the 
lead  squad  (squad  2).  The  lead  squad’s  task  is  to  conduct 
a  movement  to  contact  with  the  purpose  of  clearing  AO 
Cobra.  Their  follow-on  task  is  to  clear  AO  Cobra  for 
subsequent  UN  food  distribution  and  military  operations. 
Squad  1  ’s  task  is  to  follow  and  support  the  lead  squad  with 
the  purpose  of  clearing  AO  Cobra.  Their  follow-on  task 
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Table  1:  Input  Factors  for  the  Peace-Enforcement  Scenario 


Factor 

Name 

Description 

Range 

Frequency  Assignments 
Batch  1  Batch  2  Batch  3 

U 

Blue  Squad  1  Contact  Cohesiveness:  controls  the 
propensity  to  remain  with  the  squad  when  it  encounters 
Red  agents. 

-64,  +64 

1/81 

29/81 

10/81 

F 

Blue  Lead  Squad  Contact  Cohesiveness:  controls  the 
propensity  to  remain  with  the  squad  when  it  encounters 
Red  agents. 

-64,  +64 

4/81 

1/81 

17/81 

G 

Blue  Squad  3  Injured  Cohesiveness  controls  the 
propensity  to  remain  with  the  squad  when  one  or 
more  members  are  injured. 

-64,  +64 

10/81 

4/81 

29/81 

P 

Movement  speed  for  all  Blue  agents. 

72,  200 

17/81 

10/81 

1/81 

V 

Red  Aggression:  controls  Red’s  propensity  to  pursue 
a  perceived  threat. 

-64,  +64 

29/81 

17/81 

4/81 

Figure  2:  Initial  Graphical  Depiction  of  MANA  Peace- 
enforcement  Scenario 

is  to  clear  AO  Python.  Squad  3’s  task  is  to  follow  and 
support  the  lead  squad  with  the  purpose  of  clearing  AO 
Cobra.  Their  follow-on  task  is  to  clear  AO  Boa  (a  small 
urban  area  with  four  building  structures).  After  the  lead 
squad  clears  AO  Cobra,  the  platoon  HQ  moves  to  AO  Boa 
to  provide  supporting  fires. 

Red  has  a  five-member  element  located  near  AO  Cobra 
and  two  two-member  elements  patrolling  along  the  move¬ 
ment  routes  of  the  two  Blue  supporting  squads.  Red  also 
has  a  two-member  element  near  AO  Boa.  A  Yellow  three- 
member  element  begins  in  the  midst  of  the  Blue  forces  in 
the  upper  left  of  Figure  2.  Yellow  is  initially  non-hostile, 
but  becomes  hostile  after  discovering  that  there  is  no  potable 
water  in  the  vicinity  of  AO  Rattler.  Yellow  then  seeks  small 


arms  from  the  vicinity  of  AO  Boa  and  moves  to  the  vicinity 
of  AO  Python. 

The  devised  scenario  runs  for  a  user-specified  time  in¬ 
terval  before  terminating.  The  scenario  is  challenging  since 
the  Blue  force  is  subjected  to  a  series  of  encounters  with 
the  Red  force  and  an  originally  non-hostile  force  (Yellow) 
turns  hostile  as  the  scenario  progresses.  In  this  scenario, 
Red  is  aggressive  and  exchanges  fire  with  Blue  rather  than 
running  away.  Modifying  the  agent  personalities,  e.g.,  mak¬ 
ing  some  of  the  Red  agents  less  likely  to  approach  large 
groups  of  Blue  soldiers,  could  lead  to  another  scenario  with 
quite  different  behavior.  The  ability  to  make  such  changes 
quickly  is  one  of  the  benefits  of  an  agent-based  modeling 
platform  such  as  MANA. 

4.2  Input  Factors 

Because  our  primary  interest  is  to  determine  the  feasibility  of 
applying  FBD  in  a  data-farming  environment,  rather  than 
conduct  a  thorough  exploration  of  a  particular  scenario, 
we  vary  only  five  factors.  These  are  the  factors  Cioppa 
(2002)  found  to  be  most  influential  when  he  used  a  nearly 
orthogonal  Latin  hypercube  design  to  examine  22  factors. 
We  leave  the  remaining  factors  at  their  nominal  values  (i.e., 
the  base  settings)  in  all  runs  of  the  scenario.  Table  1  lists 
the  factors,  along  with  brief  descriptions  and  the  ranges 
over  which  they  are  varied. 

4.3  Output  Responses 

At  the  time  this  study  was  conducted,  MANA’s  outputs 
were  limited  to  the  numbers  of  agents  “killed”  during  the 
user-specified  number  of  time  steps  before  the  simulation 
terminates.  If  Blue  is  using  non-lethal  weapons,  then  the 
number  of  Red  “killed”  corresponds  to  those  incapacitated 
for  the  remainder  of  the  operation.  Wagner,  Sanders,  and 
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Mylander  (1999)  provide  guidance  on  choosing  an  appropri¬ 
ate  measure  of  effectiveness  (MOE):  it  must  be  quantitative, 
measurable,  reflect  both  the  benefits  and  penalties  of  a  par¬ 
ticular  course  of  action,  and  a  significant  increase  (decrease) 
must  correspond  to  a  significant  improvement  (worsening) 
in  achieving  the  decision-maker’s  objective.  We  choose  not 
to  limit  our  investigation  to  a  single  MOE.  Let  Ng  and  Nr 
denote  the  numbers  of  Blue  and  Red  agents  in  the  scenario. 
Our  four  performance  measures  follow. 

1 .  Kg  '-  the  number  of  Blue  killed, 

2.  Kr :  the  number  of  Red  killed, 

3.  ER  =  KR/(Kg  +  1):  the  modified  exchange  ratio, 

4.  FER  —  Kr/Kr :  the  fractional  exchange  ratio. 
Kg  and  Kr  wee  the  MANA  outputs,  while  ER  and  FER  are 
computed  from  these  output  streams.  The  denominator  in 
ER  is  Kg  +  1  instead  of  K g  because,  in  some  of  the  runs, 
no  Blue  agents  were  killed.  A  similar  adjustment  could  be 
made  for  FER,  but  was  unnecessary  for  our  scenario  since 
Red  always  sustained  losses. 

4.4  FBD  Implementation 

We  want  the  ability  to  fit  a  second-order  polynomial  meta¬ 
model  to  the  output  responses.  As  indicated  by  the  Design 
program  (Sanchez,  2002),  the  spectrum  must  be  partitioned 
into  8 1  discrete  frequencies  to  accommodate  unique  indica¬ 
tor  frequencies  for  each  of  the  5  main  effects,  5  quadratic 
terms,  and  20  effects  corresponding  to  the  10  interaction 
terms.  The  three  sets  of  frequency  assignments  are  provided 
in  the  last  three  columns  in  Table  1.  We  consider  each  set 
to  be  one  batch  of  five  hundred  “rows”  of  eighty-one  “ge¬ 
netically  engineered  strains”  that  we  plant  using  the  MANA 
distillation.  In  other  words,  the  frequency  assignments  re¬ 
main  the  same  for  all  factors  within  each  planted  batch 
of  data.  The  settings  at  the  beginning  of  each  oscillation 
are  assigned  to  their  respective  maxima  as  in  equation  (3). 
We  replicate  the  set  of  eighty-one  experimental  units  five 
hundred  times  in  each  batch  since  we  know  (from  Cioppa, 
2002)  the  results  are  highly  variable.  We  also  plant  three 
batches  of  data  in  the  data  landscape  since  we  have  three 
frequency  assignment  schemes  for  the  factors.  The  total 
number  of  experimental  units  grown  in  our  FBD  is  121,500. 
It  took  about  31  hours  to  complete  the  runs. 

4.5  Results 

We  harvest  the  output  data  sets  and  process  the  two  MOPs 
and  the  two  attrition  ratios  for  all  three  batches  through 
the  spectral  analysis  program,  Fourier.  We  then  use  the 
list  of  indicator  frequency  mappings  (provided  by  Design 
along  with  the  driving  frequencies)  to  collapse  the  result¬ 
ing  response  spectrum  by  the  corresponding  terms  in  the 
regression  model. 


Regression  Term 


Figure  3:  Signal-to-Noise  Ratio  for  Blue  Killed 


Regression  Term 

Figure  4:  Signal-to-Noise  Ratio  for  Red  Killed 

As  Step  6  in  our  procedure  describes,  we  then  map 
the  indicator  frequencies  in  each  batch  to  the  associated 
terms  in  our  regression  model.  Because  the  spectrum  is  a 
partition  of  discrete  frequency  bins,  we  present  the  spectra 
as  stacked  bar  graphs  rather  than  continuous  linear  graphs. 
The  combined  SNR  graphs  in  Figures  3-5  illustrate  the 
results  for  Kg,  Kr,  and  ER,  respectively.  The  SNR  graph 
for  FER  (not  shown)  is  quite  similar  to  that  for  ER.  The 
horizontal  lines  in  these  figures  indicate  the  critical  F- values 
for  testing  the  statistical  significance  of  the  terms.  Recall 
that  the  interaction  terms  have  twice  the  degrees  of  freedom 
as  do  the  main  and  quadratic  effects  since  there  are  two 
indicator  frequencies  for  each  interaction  term. 

The  three  shades  in  each  term  correspond  to  contribu¬ 
tions  from  the  three  different  batches.  Since  the  batches  are 
independent,  we  could  have  chosen  to  run  a  single  batch. 
(This  differs  from  the  FDE  approach,  where  the  system 
may  have  inherent  tendencies  to  dampen  or  magnify  effects 
at  specific  frequencies.)  However,  running  three  batches  is 
informative.  The  differences  in  spike  sizes  across  the  three 
batches  for  several  terms  either  indicates  that  the  system  is 
highly  variable,  or  else  highlights  the  fact  that  certain  sam- 
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Figure  5:  Signal-to-Noise  Ratio  for  Modified  Exchange 
Ratio  (Red/Blue  +1) 

pling  patterns  (frequencies)  are  more  powerful  than  others 
for  detecting  specific  types  of  effects.  Were  we  to  perform 
secondary  experiments,  we  would  continue  to  use  a  total  of 
at  least  121,500  observations  rather  than  relying  on  results 
from  a  single  batch  of  40,500  new  observations.  Recall 
that  the  5  factors  we  vary  are  those  that  Cioppa  (2002) 
found  to  have  the  strongest  effects  on  the  (unmodified)  ex¬ 
change  ratio  Kr/Kb  after  using  a  nearly-orthogonal  Latin 
hypercube  design  to  plant  the  data.  His  regression  model 
includes  main  effects  for  all  five  factors  (F,  G,  P,  U,  and 
V),  a  quadratic  term  for  U,  and  interaction  terms  for  GU 
and  FV.  (It  also  includes  four  main  effects  and  interactions 
involving  factors  we  did  not  vary.)  Figure  5  illustrates  that 
the  five  main  effects,  one  quadratic  effect,  and  two  interac¬ 
tions  identified  by  Cioppa  were  also  statistically  significant 
in  our  FBD.  We  also  found  that  G  and  V  have  quadratic 
effects  on  ER,  and  that  other  interaction  terms  are  present. 
It  is  not  surprising  that  our  results  are  somewhat  different. 
We  varied  fewer  factors,  modified  the  performance  measure 
slightly,  and  used  different  ranges  (in  part  to  avoid  a  range 
of  movement  speeds  where  he  found  that  the  simulation 
model  was  broken). 

Our  results  also  highlight  the  importance  of  examining 
several  different  performance  measures  if  the  choice  of 
MOE  is  unclear.  Factor  V  (Red  aggression)  is  by  far  the 
dominant  factor  in  the  SNR  for  Kb  (Figure  3).  However, 
the  SNR  for  K r  is  quite  different  (Figure  4),  with  V  having 
the  second-lowest  main  effect  among  the  five  factors.  The 
SNR  for  Kr  also  indicates  that  a  number  of  interaction 
terms  involving  F,  G,  P  and  U  contribute  as  much  or  more 
to  the  variability  in  K r  as  does  factor  V .  The  SNRs  for  ER 
and  PER  show  F  has  the  largest  main  effect,  U  and  V  have 
similar  main  effects,  and  U  has  a  very  strong  quadratic  effect 
as  well.  Other  terms  are  statistically  significant,  but  these 
four  account  for  the  vast  majority  of  the  variability  in  Y. 
So,  an  analyst  using  Kb  as  the  performance  measure  might 
conclude  that  since  Red  aggression  is  the  primary  driver. 


there  is  little  that  can  be  done  in  planning  the  operation  to 
reduce  the  likelihood  of  Blue  losses.  An  analyst  focusing 
on  Red  losses  would  see  these  are  affected  by  Blue’s  contact 
cohesiveness  (factors  U  and  F)  and  Blue’s  movement  speed 
(factor  P).  In  both  cases,  the  number  killed  for  one  affiliation 
is  a  function  of  the  other  side’s  actions,  not  their  own. 
However,  when  looking  at  the  attrition  ratios,  characteristics 
of  squads  from  both  sides  contribute  to  the  outcome. 

Since  the  spectral  terms  are  proportional  to  sums  of 
squared  metamodel  coefficients,  further  analysis  would  be 
needed  to  determine  appropriate  levels  for  maximizing  or 
minimizing  the  various  performance  measures.  This  can  be 
accomplished  using  multiple  regression  without  generating 
additional  data.  For  numerical  stability,  it  is  best  to  use 
standardized  independent  variables  X*  where 

X*  —  a i  cos (a>it)  t  =  1, . . . ,  N 

(or  X*  =  cos  (cot  f ) )  for  i  =  1, . . . ,  k.  Quadratic  and  inter¬ 
action  explanatory  variables  can  be  obtained  by  multiplying 
the  appropriate  X* . 

5  CONCLUDING  REMARKS 

We  have  presented  a  frequency-based  design  (FBD)  ap¬ 
proach  that  can  be  used  to  identify  important  determinants 
of  the  performance  of  terminating  simulations.  Our  work 
was  motivated  by  the  need  to  develop  efficient,  effective 
tools  for  planting  data  in  a  data-farming  environment.  As  a 
proof  of  concept  for  the  potential  utility  of  this  approach,  we 
illustrated  FBDs  using  an  agent-based  model  of  a  peace- 
enforcement  operation.  Our  results  were  in  consonance 
with  those  obtained  in  an  earlier  (and  broader)  investigation 
that  used  the  exchange  ratio  as  the  performance  measure. 
We  also  considered  three  other  performance  measures,  and 
found  that  looking  at  multiple  performance  measures  gave 
additional  insights  into  the  relative  importance  of  the  fac¬ 
tors.  Therefore,  we  concluded  that  FBD  is  not  only  a 
feasible  method  for  data  farming,  but  also  a  useful  tech¬ 
nique  for  factor  screening  that  is  easy  to  generate.  We 
are  currently  examining  several  issues  regarding  FBDs  in 
order  to  further  increase  their  utility.  First,  we  are  look¬ 
ing  at  sequential  analysis  and  display  of  the  results.  This 
may  be  important  either  when  the  individual  simulation 
runs  take  more  time,  or  when  we  the  number  of  factors  is 
large.  Information  obtained  early  on  the  experiment  may 
allow  the  analyst  greater  flexibility  in  changing  the  driving 
frequencies  and/or  factor  ranges  as  initial  results  indicate 
either  very  strong  or  very  weak  effects.  Second,  Wu  (2002) 
considered  ways  of  sonifying  the  output.  While  further 
work  is  needed,  sonification  may  be  beneficial  because  it  is 
another  channel  of  information  that  could  augment  visual 
displays.  Finally,  we  are  assessing  the  use  of  expert  opinion 
about  what  factors  are  likely  to  be  the  most  important  in 
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assigning  driving  frequencies.  Our  goals  are  to  see  whether 
this  additional  information  can  be  exploited  to  reduce  the 
data  requirements. 
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